Add native secret-store config resolution - #1036

Summary

Moves static app-config credentials from plaintext blob values to secret-store key
references resolved after envelope verification, and fixes the logical-to-physical
store mapping that broke the Fastly deployment. The core design is sound: integrity
verification genuinely precedes resolution, resolution is atomic (the blob is left
untouched on failure), the deploy/load validation split keeps value checks on the load
path where PartnerRegistry::from_config still fails closed, and the two end-to-end
payload tests cover both the all-credentials-resolve and inactive-feature-skip arms.

Four blocking items: resolution discards the one diagnostic that would explain a
mis-mapped store, the documented migration order opens a total outage window, the
Fastly Hooks::routes() path reads the store mapping from the wrong source, and the
EdgeZero dependency is pinned to an unmerged upstream commit.

3 of the inline comments below carry a one-click GitHub suggestion — use
Commit suggestion (or Add suggestion to batch) to apply them as commits on
the PR branch. The remaining comments describe the fix in prose because the change
spans multiple files, needs a new import, or adds code outside the diff. No
suggestion in this review was scratch-verified — local runs were skipped for this
pass, so please re-run the matching checks after applying.

Blocking

🔧 wrench

• Secret-store resolution throws away every adapter's diagnostic — see inline at crates/trusted-server-core/src/secret_resolution.rs:164
• Documented migration order opens a full outage window — see Cross-cutting below
• Hooks::routes() reads the wrong source for the store mapping — see inline at crates/trusted-server-adapter-fastly/src/app.rs:1261

❓ question

• EdgeZero pinned to an unmerged upstream PR — see Cross-cutting below

Non-blocking

♻️ refactor / 🤔 thinking / ⛏ nitpick / 🌱 seedling

• Required S3 secret references still have serde defaults — see inline at crates/trusted-server-core/src/settings.rs:767
• Feature-enablement logic duplicated in three places — see inline at crates/trusted-server-core/src/config_payload.rs:63
• EchoSecretStore makes resolution untestable — see inline at crates/trusted-server-core/src/config_payload.rs:145
• expect() on the Tinybird token traps the Wasm guest — see inline at crates/trusted-server-adapter-fastly/src/tinybird.rs:57
• Deploy validation misses duplicate partner key names — see inline at crates/trusted-server-core/src/ec/registry.rs:74
• New docs bullets lost their markdown hard breaks — see inline at docs/guide/configuration.md:1620
• partners = [] is redundant and a footgun — see inline at trusted-server.example.toml:17
• Two overlapping ways to express leaf optionality — see inline at crates/trusted-server-core/src/secret_resolution.rs:64
• Spin's five declared secret variables read as a contract — see inline at crates/trusted-server-adapter-spin/spin.toml:28

Cross-cutting / body-level findings

• 🔧 Documented migration order opens a full outage window — docs/guide/configuration.md:60-72 gives the order: populate store, replace values with key names, ts config validate + ts config push, then "restart/redeploy instances as needed."

  Step 3 lands the reference-bearing blob while the old binary is still serving. On Fastly each request reads the config store fresh, so from that instant every request runs Ec::validate_passphrase — which requires at least 32 bytes on main today (MIN_PASSPHRASE_LENGTH = 32, crates/trusted-server-core/src/settings.rs) — against passphrase = "ec_passphrase" (13 bytes). That yields short_passphrase, config load fails, and the service returns its startup-error response for all traffic until the redeploy finishes.

  The reverse mismatch fails too: a new binary reading a plaintext blob resolves each plaintext secret as a key name. There is no safe intermediate state — the binary and the blob have to flip together, and the doc currently puts the break in the middle. Please correct the ordering and add an explicit warning that a mismatched binary/blob pair fails config load outright. The staged Fastly deployment cited in the PR description would not surface this, since no old binary is in play there.

• ❓ EdgeZero pinned to an unmerged upstream PR — Cargo.toml:57-62 moves all six edgezero crates from stable tag v0.0.4 to git rev 0d6ebf9b0250efa5f7031a93ec7b7f09f2c9bf34, a commit on the still-open stackpop/edgezero#344. Merging this puts main on a branch commit of an unmerged PR: if #344 is rebased or force-pushed before it merges, that commit can become unreachable and main stops building.

  You disclosed this in the PR description, and the issue comment suggests this lands in #1019 first, so this may already be handled. It still needs an explicit answer because it constrains main: hold this PR until #344 merges and re-pin to a tag, or is a rev pin on main acceptable here?

• 📝 CI coverage gap on the reviewed head — only Analyze (javascript-typescript) ran on 1315cdb1. The full gate suite (cargo fmt/test/clippy, all four adapters, cross-adapter parity, vitest, format-docs, integration and browser tests) last ran green on the merge commit 598f7100, three commits earlier. That leaves 070397f1 Resolve static credentials through typed config, b1e967e3, and 1315cdb1 without Rust, adapter, or lint coverage. Worth re-triggering the suite on the current head before merge, independent of the findings above.

• 👍 validation_error_summary is a real leak fix — crates/trusted-server-core/src/settings.rs:2387-2424 walks ValidationErrors emitting only path: code, never validator's params, which hold the offending value. The previous code formatted ValidationErrors wholesale into a config error message.

• 👍 Deleting S3_CREDENTIALS_CACHE removes a genuinely bad structure — crates/trusted-server-core/src/proxy.rs previously kept a process-global HashMap keyed on the plaintext secret access key, with unbounded growth and a poisoning-prone Mutex. Startup-resolved values are strictly better.

• 👍 IntegrationSettings's custom Debug closes the DataDome-key leak that the flattened JsonValue map would otherwise print.

• 👍 The two payload resolution tests are the right pair — resolves_all_static_credentials_from_the_mapped_default_store proves every path arm resolves through a mapped physical store, and inactive_optional_features_do_not_resolve_stale_secret_references proves disabled features do not demand stale references. Also good: dropping include_str!("trusted-server.example.toml") from the Spin and Cloudflare startup paths in favour of a hard error.

CI Status

• Analyze (javascript-typescript): PASS
• cargo fmt: not run on this head (PASS on 598f7100)
• cargo test: not run on this head (PASS on 598f7100)
• cargo test (axum native): not run on this head (PASS on 598f7100)
• cargo test (cross-adapter parity): not run on this head (PASS on 598f7100)
• cargo test (ts CLI, native): not run on this head (PASS on 598f7100)
• cargo check (cloudflare native + wasm32-unknown-unknown): not run on this head (PASS on 598f7100)
• cargo check/build/test (spin native + wasm32-wasip1): not run on this head (PASS on 598f7100)
• integration tests: not run on this head (PASS on 598f7100)
• integration tests (Fastly EC lifecycle): not run on this head (PASS on 598f7100)
• browser integration tests: not run on this head (PASS on 598f7100)
• prepare integration artifacts: not run on this head (PASS on 598f7100)
• vitest: not run on this head (PASS on 598f7100)
• format-typescript: not run on this head (PASS on 598f7100)
• format-docs: not run on this head (PASS on 598f7100)
• Analyze (rust): not run on this head (PASS on 598f7100)
• Analyze (actions): not run on this head (PASS on 598f7100)
• CodeQL: not run on this head (PASS on 598f7100)

No check reported a fail or cancel bucket. Branch protection reported no required checks for this PR.

Summary

Well-executed change: the resolution model (verify envelope, strip inactive references, resolve, validate runtime settings) is fail-closed, the push-time/runtime validation split is coherent across all four adapters, and the test coverage in config_payload.rs and secret_resolution.rs is thorough. Two blocking findings: a secret-exposure path in the resolution-failure error message, and the failed CodeQL check.

4 of the inline comments below carry a one-click GitHub suggestion — use Commit suggestion (or Add suggestion to batch) to apply them as commits on the PR branch. The remaining comments describe the fix in prose because the change touches multiple locations and can't be auto-applied.

Blocking

🔧 wrench

• Resolution-failure error can log a plaintext secret from a legacy blob — see inline at crates/trusted-server-core/src/secret_resolution.rs:169
• CodeQL check failed: 15 high rust/cleartext-logging alerts — see Cross-cutting below

Non-blocking

♻️ refactor / 🤔 thinking / ⛏ nitpick / 📝 note / 🌱 seedling

• .env.example ships the Fastly store mapping active, breaking the documented Axum flow — see inline at .env.example:11 (suggestion)
• Migration guide doesn't warn that previously legal short secrets now fail startup — see inline at docs/guide/configuration.md:70 (suggestion)
• Missing required leaf reports "must be a string" instead of "missing" — see inline at crates/trusted-server-core/src/secret_resolution.rs:151 (suggestion)
• server_side_key_secret_name holds the resolved key value at runtime — see inline at crates/trusted-server-core/src/integrations/datadome.rs:184 (suggestion)
• validate_config_for_deploy uses HashMap<_, ()> as a set — see inline at crates/trusted-server-core/src/ec/registry.rs:77
• Hooks::stores() duplicates edgezero.toml — see inline at crates/trusted-server-adapter-fastly/src/app.rs:1332
• Cargo.lock rewrote prost's itertools edges — see Cross-cutting below

Cross-cutting / body-level findings

• 🔧 CodeQL check failed: 15 high rust/cleartext-logging alerts. Not required under branch protection, but a CI gate this repo treats as blocking. I inspected all 15: they are taint over-approximation — CodeQL now treats everything flowing out of resolve_secret_references / validate_tinybird_secret / validate_admin_handler_passwords as secret-tainted and flags logs of plainly non-secret fields (asset-route prefixes in settings.rs, DataDome registration flags in datadome.rs:979, consent clamping in consent_config.rs, header names in response_privacy.rs, etc.). No alert is a real value leak — the nearest real vector is the inline finding at secret_resolution.rs:169. The alerts still need triage: dismiss each in the code-scanning UI with a justification (or add a CodeQL model/sanitizer exclusion), otherwise this check stays red here and re-fires on every future PR touching these paths.
• ⛏ Cargo.lock moved prost's itertools dependency edges from 0.13.0 to 0.10.5. The edgezero pin update also rewrote prost-build/prost-derive's itertools edges down to the already-present 0.10.5 while 0.13.0 stays in the graph for other consumers — unintended churn from edge unification during the scoped update. Consider hand-restoring the 0.13.0 edges so the lock diff stays scoped to the edgezero bump.

CI Status

• browser integration tests: PASS
• integration tests (Fastly EC lifecycle): PASS
• integration tests: PASS
• CodeQL: FAIL
• cargo test (ts CLI, native): PASS
• cargo test (cross-adapter parity): PASS
• cargo check/build/test (spin native + wasm32-wasip1): PASS
• cargo check (cloudflare native + wasm32-unknown-unknown): PASS
• format-docs: PASS (required)
• cargo test: PASS (required)
• cargo test (axum native): PASS
• format-typescript: PASS (required)
• Analyze (rust): PASS
• Analyze (javascript-typescript): PASS
• Analyze (javascript-typescript): PASS
• cargo fmt: PASS (required)
• prepare integration artifacts: PASS
• vitest: PASS
• Analyze (actions): PASS

Summary

Second pass, reviewing head 76f6f13. The feedback commit addresses every finding from the previous review: the resolution-failure error now drops both the key name and the underlying platform error (with a regression test asserting a legacy plaintext value never reaches diagnostics), the new 32-byte minimums were removed in favor of pre-PR behavior (bypass-credential strength enforcement moved back to request time, with a request-level test), .env.example no longer ships the Fastly mapping active, and the stores() metadata is now pinned to edgezero.toml by a manifest-parsing test. What remains blocking is the open CodeQL alert set; one stale doc claim and the lockfile nit round out the list.

1 of the inline comments below carries a one-click GitHub suggestion — use Commit suggestion to apply it as a commit on the PR branch.

Blocking

🔧 wrench

• CodeQL: 15 high rust/cleartext-logging alerts still open — see Cross-cutting below

Non-blocking

⛏ nitpick

• Stale "must be at least 32 bytes" claim for proxy_secret — see inline at docs/guide/configuration.md:361 (suggestion)
• Cargo.lock prost itertools edges still rewritten — see Cross-cutting below

Cross-cutting / body-level findings

• 🔧 CodeQL: 15 high rust/cleartext-logging alerts still open. Carried over from the previous review round. The code fix in 76f6f13 does not clear them — all 15 are taint over-approximation (CodeQL treats everything flowing out of resolve_secret_references / validate_tinybird_secret / validate_admin_handler_passwords as secret-tainted and flags logs of plainly non-secret fields), and all 15 remain open on this PR, so the CodeQL check will fail again once analysis reruns on this head. They need triage: dismiss each in the code-scanning UI with a justification, or add a CodeQL suppression/model exclusion — otherwise this check stays red here and re-fires on every future PR touching these paths.
• ⛏ Cargo.lock still carries the rewritten prost itertools edges (0.13.0 → 0.10.5 while 0.13.0 stays in the graph for other consumers) — unaddressed nit from the previous review; the earlier inline thread on this stays open, so no new inline comment here. Hand-restoring the 0.13.0 edges keeps the lock diff scoped to the edgezero bump.

CI Status

GitHub checks have not yet run for head 76f6f13 — only one check has reported; everything else is pending/not started. Local verification was run in the reviewer worktree at this head instead: cargo fmt --all -- --check, cargo clippy-fastly, targeted cargo test-fastly for the modules this head touches (10 secret_resolution + 15 config_payload + 67 datadome + 43 registry + the fastly manifest-metadata test), and prettier for the changed docs — all pass.

• Analyze (javascript-typescript): PASS
• CodeQL: not run on this head (15 alerts from the prior analysis remain open)
• browser integration tests: not run
• integration tests (Fastly EC lifecycle): not run
• integration tests: not run
• cargo test (ts CLI, native): not run
• cargo test (cross-adapter parity): not run
• cargo check/build/test (spin native + wasm32-wasip1): not run
• cargo check (cloudflare native + wasm32-unknown-unknown): not run
• format-docs: not run (required; passes locally)
• cargo test: not run (required; touched modules pass locally)
• cargo test (axum native): not run
• format-typescript: not run (required)
• Analyze (rust): not run
• cargo fmt: not run (required; passes locally)
• prepare integration artifacts: not run
• vitest: not run
• Analyze (actions): not run

Summary

Adds secret-store reference resolution to app config: the blob carries key names, and each adapter resolves them into redacted runtime values after integrity verification. The core mechanism is well built and genuinely fail-closed — one central resolution point, envelope.verify() strictly before resolution on all four adapters, clone-then-swap so a partial resolution never reaches Settings, empty resolved values rejected centrally, no secret in any error message, and zero per-request app-config secret reads left anywhere. The api_token: Option auth change is fail-closed and verified from both directions. The Fastly logical→physical bug is genuinely fixed on the startup path and all three reload paths.

The blockers are concentrated in the operator-facing surface rather than the resolution logic. Three of them produce an outage or a dead local environment for anyone following the instructions as written.

10 of the inline comments below carry a one-click suggestion — use Commit suggestion (or Add suggestion to batch) to apply them. Every suggestion was applied in a scratch worktree at this head and verified: cargo fmt --all -- --check clean, clippy-fastly / clippy-axum / clippy-cloudflare / clippy-spin-native all clean at -D warnings, 2,477 tests passing across test-fastly (including 2,284 core tests), test-axum, test-cloudflare and test-spin, pinned prettier clean on both changed docs, and both changed TOML files re-parsed. The remaining comments describe the fix in prose because the change spans multiple files, needs a validator split first, or targets lines outside a RIGHT-side diff hunk.

Blocking

🔧 wrench

• KeyInNamedStore fields would silently resolve from the default store — see inline at crates/trusted-server-core/src/secret_resolution.rs:29
• trusted_client_ip.shared_secret is the one credential left plaintext in the blob — see inline at crates/trusted-server-core/src/config.rs:135
• Local fastly compute serve cannot start — no app-config secrets seeded — see inline at fastly.toml:62
• Migration runbook never creates or links the physical store — see inline at docs/guide/configuration.md:70
• Deploy path repeats the same omission — see inline at docs/guide/getting-started.md:160
• Axum quick-start cannot complete — starter config ships reserved placeholder domains — see inline at .env.dev:5 and docs/guide/getting-started.md:73
• Stale guardrail claim: deploy validation no longer rejects a placeholder handler password — see inline at trusted-server.example.toml:41
• CodeQL gate is red — see Cross-cutting below
• PR is CONFLICTING; two conflicts are competing designs — see Cross-cutting below

❓ question

• Spin hardcodes the config-store name, contradicting the docs this PR adds — see inline at crates/trusted-server-adapter-spin/src/app.rs:61
• Unrelated Cargo.lock churn: prost's itertools 0.13.0 → 0.10.5 — see inline at Cargo.lock:3679
• PR description is inaccurate in two places — see Cross-cutting below

Non-blocking

🤔 thinking / ♻️ refactor / 🏕 camp site / ⛏ nitpick

• Rollback window: an old binary uses the documented key name as a live HMAC key — see inline at crates/trusted-server-core/src/config.rs:136
• A suppressed telemetry error became a per-request 500 — see inline at crates/trusted-server-adapter-fastly/src/tinybird.rs:60
• validate_config_for_startup / _for_deploy are byte-identical, so resolved_secrets is a no-op — see inline at crates/trusted-server-core/src/config.rs:284
• pull_sync_enabled read with as_bool() but deserialized with from_value_or_str — see inline at crates/trusted-server-core/src/config_payload.rs:84
• resolve_leaf discards the PlatformError cause, losing the primary triage signal — see inline at crates/trusted-server-core/src/secret_resolution.rs:167
• No test that a required reference failing lookup fails the load — see inline at crates/trusted-server-core/src/config_payload.rs:455
• ts_pull_token's requirement has zero coverage on either side — see inline at crates/trusted-server-core/src/ec/registry.rs:403
• require_nonempty_token is a flag that earns nothing — see inline at crates/trusted-server-core/src/ec/registry.rs:348
• Cloudflare re-resolves every secret per request, against request #1's Env — see inline at crates/trusted-server-adapter-cloudflare/src/app.rs:51
• Cloudflare is the only adapter with no store mapping and no comment saying why — see inline at crates/trusted-server-adapter-cloudflare/src/app.rs:130
• spin.toml's request-signing config variables are now silently inert — see inline at crates/trusted-server-adapter-spin/src/platform.rs:127
• EDGEZERO__STORES__SECRETS__…__NAME silently redirects the Axum lookups — see inline at docs/guide/getting-started.md:104
• Only worked example puts the passphrase in argv — see inline at docs/guide/fastly.md:326
• Five secret key names live in five files with nothing keeping them in sync — see inline at crates/trusted-server-integration-tests/fixtures/configs/trusted-server.integration.toml:4
• validation_error_summary drops the validator message — see inline at crates/trusted-server-core/src/settings.rs:3403
• Dead deprecation branches in try_new — see inline at crates/trusted-server-core/src/integrations/datadome.rs:390
• S3Credentials rebuilt with three allocations per signing call — see inline at crates/trusted-server-core/src/proxy.rs:878
• Axum reaches for a fully-qualified path instead of the import above it — see inline at crates/trusted-server-adapter-axum/src/app.rs:68
• pub mod secret_resolution should be pub(crate) — see inline at crates/trusted-server-core/src/lib.rs:67
• tinybird.remove("access_token_secret") is dead — see inline at crates/trusted-server-core/src/config_payload.rs:73
• Duplicate use lines — see inline at crates/trusted-server-core/src/settings_data.rs:6
• Two assertions without messages — see inline at crates/trusted-server-adapter-fastly/src/app.rs:1432
• EcPartner::api_token doc still says "Plaintext API token" — see inline at crates/trusted-server-core/src/settings.rs:377
• .env.example ordering reads backwards — see inline at .env.example:9
• Real deployed Fastly hostname retained — see inline at docs/guide/ec-setup-guide.md:31

📝 note

• Severity change: DataDome went from request-time fail-open to boot-time fail-closed — see inline at crates/trusted-server-core/src/integrations/datadome/protection.rs:86
• Trimming the resolved bypass credential invalidates whitespace-carrying values — see inline at crates/trusted-server-core/src/integrations/datadome.rs:408
• Every Fastly request now opens a config store and makes ~10 dictionary reads — see inline at crates/trusted-server-adapter-fastly/src/main.rs:93
• stores().kv is declared but nothing on the Fastly path consumes it — see inline at crates/trusted-server-adapter-fastly/src/app.rs:1338
• jq dependency not in Prerequisites — see inline at docs/guide/getting-started.md:80
• configuration.md S3 table vs example key mismatch — see inline at docs/guide/configuration.md:1099

👍 praise

• Hand-written IntegrationSettings Debug closes a real leak — see inline at crates/trusted-server-core/src/settings.rs:220
• hooks_store_metadata_matches_edgezero_manifest pins exactly the right invariant — see inline at crates/trusted-server-adapter-fastly/src/app.rs:1394
• Order of operations is right and centrally enforced — see inline at crates/trusted-server-core/src/config_payload.rs:38

Cross-cutting / body-level findings

• 🔧 CodeQL gate is red — 15 open rust/cleartext-logging high alerts. Not in branch protection's required set, so not merge-blocking mechanically. All 15 read as false positives: validate_tinybird_secret (settings.rs:1971) formats only "{setting} must be non-empty after secret resolution" — the setting name, no value; the flagged sinks log route.prefix, a cache rule id, a path, and DataDome's sdk_origin/rewrite_sdk/enable_protection; the named source try_new_with_secret_validation does not exist anywhere in the tree; and 9 of the 15 sinks are in files this PR does not touch (consent_config.rs, storage/kv_store.rs, response_privacy.rs, auction/orchestrator.rs, management_api.rs, axum/src/platform.rs). Root cause is field-insensitive taint: Settings now carries resolved plaintext secrets, so every log of any Settings-derived field lands in a taint path. The architectural signal is real and new even though each alert is not — before this PR, "log a Settings field" could not leak a credential. Resolve by dismissing the 15 with a written justification, so the next true positive in this rule is visible again; or better, give resolved secrets a wrapper whose Debug/Display/Serialize cannot emit the value. Note Redacted is serde-transparent, so it does not close the serialize half.

• 🔧 PR is CONFLICTING; two of the six conflicts are competing designs. git merge-tree origin/main conflicts in Cargo.lock, Cargo.toml, crates/trusted-server-adapter-fastly/src/app.rs, crates/trusted-server-adapter-fastly/src/main.rs, crates/trusted-server-core/src/config.rs and docs/guide/configuration.md. In config.rs, main already carries a secret_fields() stub returning Vec::new() whose comment defers secret-store references "plus operator migration work tracked separately" — worth confirming that deferred work is what this PR delivers. In the Fastly adapter, main threads &EnvConfig with config_store_name(env) / config_key(env), while this PR replaces that seam with RuntimeStoreConfig / DEFAULT_CONFIG_STORE_ID / env_config_from_runtime_dictionary. Separately, this PR rewrites "EdgeZero's env overlay" → "The pinned EdgeZero loader" in three places in configuration.md; that phrasing only holds while pinned to a rev and goes stale as soon as the pin returns to a tag.

• ❓ The PR description is inaccurate in two places. (1) It says generate-viceroy-config.rs "Generate[s] local secret-store data for references found in integration configuration." It does not — build_app_config_envelope (lines 113-134) only swaps Settings::from_toml + validate_settings_for_deploy for toml::from_str::<TrustedServerAppConfig> + TrustedServerAppConfig::new, and generated_config_store_blocks (148-156) still emits only the config-store block. Every secret value is hand-maintained. (2) It describes a "signed configuration blob"; envelope.verify() is a self-computed canonical SHA-256 (edgezero-core/src/blob_envelope.rs:87-99), not a signature. The ordering is right and it is the correct primitive for the chunked Fastly path, but it defends against truncation and corruption — not against someone with config-store write access. config_payload.rs:25-26 already says "integrity verification", which is the accurate wording.

• 📝 The EdgeZero pin is a commit that exists on no upstream branch or tag. Cargo.toml:57-62 pins all six edgezero crates to rev 0d6ebf9b0250efa5f7031a93ec7b7f09f2c9bf34. Verified against stackpop/edgezero: it is not an ancestor of origin/main, git branch -a --contains and git tag --contains are both empty, and it is reachable only as a bare refs/commit/<sha>. It is 140 commits behind v0.0.7 (5c9886e5), which main now uses — so merging as-is downgrades EdgeZero across adapter-{axum,cloudflare,fastly,spin}, cli and core. Upstream stackpop/edgezero#344 is still open with mergeCommit: null, and its current head is 055f7e94, from which the pinned rev has also diverged (behind 140, ahead 12) — the branch was rebased since. If #344 lands squashed or rebased, this SHA becomes unreferenced and can be garbage-collected, at which point cargo fetch fails on every cold cache including CI. All seven edgezero packages in Cargo.lock do move consistently, so nothing is left behind on the old tag. Flagged as informational — the merge-ordering call belongs to the author and the release owner.

• 🤔 Viceroy "Option A" in getting-started.md:46-62 gained a required setup step documented nowhere. That section is untouched by this PR and still reads fastly compute serve with no config or secret preparation, but local serve now requires hand-seeded secrets (see the fastly.toml comment). grep -rn "local_server.secret_stores" docs/ returns only key-rotation.md and request-signing.md, both about signing_keys. Option B got a full rewrite; Option A got nothing. The recipe already exists at scripts/template-cache-local-test.sh:227-243. Body-level because line 46 is outside every RIGHT-side hunk.

• ♻️ resolve_secret_references deep-clones the whole config for an atomicity guarantee no caller uses (secret_resolution.rs:29 and :43). The only production caller is config_payload.rs:53, which passes a local data that is dropped on the error path anyway; the clone's sole beneficiary is does_not_mutate_data_when_resolution_fails (:396-407). It costs a full deep copy of the config JSON per instance boot inside Wasm. Resolving in place is equivalent for every real caller. Body-level because it targets the same lines as the KeyInNamedStore suggestion.

• ⛏ Stale S3Credentials doc claims a runtime store read and a deleted cache (crates/trusted-server-core/src/s3_sigv4.rs:34-36). Both claims are now false: apply_asset_origin_auth builds these from already-resolved config, and S3_CREDENTIALS_CACHE was removed by this PR. s3_sigv4.rs is not in the diff, so this cannot be an inline comment. Proposed replacement:

  /// Values are already resolved from the app-config secret store when settings are
  /// built, so the caller passes them straight through without a runtime store read.
  /// Temporary credentials can include a session token, which becomes the signed
  /// `x-amz-security-token` header.
  

• ⛏ Struct doc at settings.rs:355-356 is now false. "the plaintext is never stored at runtime" holds for PartnerConfig (hash only) but not for EcPartner.api_token, which holds the resolved plaintext in Settings for the process lifetime. Worth narrowing to "the registry stores only the hash". Outside a hunk, so body-level.

• 📌 Pre-existing real-world values in fastly.toml, all outside every diff hunk. fastly.toml:4 authors = ["jason@stackpop.com"], :10 service_id = "dysUw6h73VzeomD61eal85", and :50 data = "NVnTYrw5xoyTJDOwoUWoPJO3A6UCCXOJJUzgGTxxx7k=" — a base64 32-byte Ed25519-shaped value in signing_keys whose embedded xxx suggests deliberate mangling. viceroy-template.toml:55-59 carries an explicit "generated for testing, never used in production" attestation for the same value; fastly.toml has none. Not asked for in this PR, but a secrets-hardening PR is the natural place to file it.

• 👍 Verified clean and worth stating explicitly, so it is clear these were checked rather than skipped: verification-before-resolution ordering on all four adapters; a swallowed-failure scan of every added line (ok(), unwrap_or_default(), unwrap_or(false), let _ = — the only hits are two intentional OnceCell::set calls); per-request versus startup secret reads (the whole per-request class is closed, and the only remaining secret_store() callers read the separate request_signing.secret_store_id); the api_token: Option auth change traced from both directions; the deploy/runtime validation split, where runtime is a strict superset for value checks and a net tightening versus main; the legacy-selector compatibility bridge (all three warn, none survives a round trip, and init_cli_logger sets LevelFilter::Info so the warnings are actually visible); recursive resolution edge cases (empty arrays, Option containers, null versus absent, the internally-tagged AssetOriginAuth, non-string leaves, and no overlapping SecretField paths); spin.toml's secret-variable encoding byte-for-byte against spin_secret_variable_name; the Wrangler CI binding names and their fictional values; naming drift across code and all five manifests; WASM gating on every new item; secret exposure via every runtime Settings serialization path; and scripts/template-cache-local-test.sh end to end including its CI greps. No real secrets or real-world values are introduced anywhere by this PR, and this PR structurally reduces the tracked-fastly.toml secret-leak risk.

Recommendation

Hold. The resolution mechanism is sound and the auth change is correct — the work needed is on the operator-facing surface: the three docs/local-dev blockers, the KeyInNamedStore guard, and either closing the trusted_client_ip.shared_secret gap or stating the deferral explicitly. The merge will also need a decision on the RuntimeStoreConfig-versus-&EnvConfig seam against main.

CI Status

• browser integration tests: PASS
• integration tests: PASS
• integration tests (Fastly EC lifecycle): PASS
• prepare integration artifacts: PASS
• CodeQL: FAIL
• Analyze (rust): PASS
• Analyze (actions): PASS
• Analyze (javascript-typescript): PASS
• cargo test: PASS (required)
• cargo test (axum native): PASS
• cargo test (ts CLI, native): PASS
• cargo test (cross-adapter parity): PASS
• cargo check (cloudflare native + wasm32-unknown-unknown): PASS
• cargo check/build/test (spin native + wasm32-wasip1): PASS
• cargo fmt: PASS (required)
• format-typescript: PASS (required)
• format-docs: PASS (required)
• vitest: PASS

Summary

Third pass, reviewing head 4258a6b. The branch now merges main through #1077 and adds two substantive changes, both reviewed in full: the pass-2 docs suggestion applied verbatim, and a tip commit that integrates the newly-merged [trusted_client_ip] feature into the secret-reference model and bumps EdgeZero to v0.0.8. The new work held up under scrutiny — no new findings. What remains blocking is the open CodeQL alert set, carried over a second time.

Verified on the tip commit specifically:

• trusted_client_ip.shared_secret is wired end-to-end: required leaf under an optional section, push-time reference check, and the ≥32-graphic-ASCII validator split onto the field leaf so it prunes at push time (proven by the new test using a 31-char key name) while still enforcing on the resolved value at runtime. A present section with a missing or unresolvable key fails closed with the path-only error message.
• The json_bool_or_string_is_true fix for pull_sync_enabled matches from_value_or_str's bool::from_str exactly, so there is no accepted spelling the strip logic misses; tinybird.enabled and the DataDome flags are strict bools that fail deserialization loudly, so they need no equivalent.
• The DataDome bypass credential is no longer normalized (restoring legacy request-time semantics, with a preservation test); the server-side key keeps its legacy trim — per-field legacy behavior, deliberate.
• Merge resolutions for #1048/#1070/#1077 are correct; notably, client-IP resolution runs before header sanitization, and when app state fails to build the resolver receives no config and falls back to the untrusted peer address.

Blocking

🔧 wrench

• CodeQL: 14 high rust/cleartext-logging alerts still open — see Cross-cutting below

Non-blocking

⛏ nitpick

• Cargo.lock prost itertools edges still rewritten — see Cross-cutting below

Cross-cutting / body-level findings

• 🔧 CodeQL: 14 high rust/cleartext-logging alerts still open. Carried over from the two previous review rounds. One of the original 15 is now marked fixed; the remaining 14 are open and none were dismissed, so the CodeQL check will fail again once analysis reruns on this head. As established in the first round, all of them are taint over-approximation (Settings-derived values flowing through the resolution/validation functions into logs of plainly non-secret fields), not real value leaks — but they still need triage: dismiss each in the code-scanning UI with a justification, or add a CodeQL suppression/model exclusion, otherwise the check stays red here and re-fires on every future PR touching these paths.
• ⛏ Cargo.lock still carries the rewritten prost itertools edges (0.13.0 → 0.10.5 while 0.13.0 stays in the graph for other consumers) — unaddressed nit; the inline thread from the first review round stays open, so no new inline comment here.

CI Status

GitHub checks have not yet started for head 4258a6b (the latest complete runs — Run Format, Run Tests, Integration Tests, CodeQL Advanced, all green at the workflow level — are on the preceding merge commit 71df1aa). Local verification was run in the reviewer worktree at this head instead: cargo fmt --all -- --check, cargo clippy-fastly, cargo check-axum / cargo check-cloudflare / cargo check-spin under the EdgeZero v0.0.8 bump, targeted cargo test-fastly for the modules the tip commit touches (22 trusted_client_ip + 10 secret_resolution + 19 config_payload + 68 datadome + the fastly manifest-metadata test), cargo test-spin for the variable-encoder tests, and prettier for the changed docs — all pass.

• Run Format: not started on this head (passes locally)
• Run Tests: not started on this head (touched modules pass locally)
• Integration Tests: not started on this head
• CodeQL: not started on this head (14 alerts from the prior analysis remain open)

Summary

This lands the #846 secret-store migration cleanly: app config carries stable secret-store key names, resolution happens once after envelope verification, and runtime code no longer reads static credentials per request. The metadata contract, the deploy/runtime validation split, and the fail-closed test coverage are all well built.

One blocking problem: the tip commit (4258a6bf) bumped the EdgeZero pin from rev 0d6ebf9b to tag v0.0.8, and v0.0.8 changed how runtime_env_config reads the edgezero_runtime_env store. Rev 0d6ebf9b read unscoped EDGEZERO__* keys; v0.0.8 reads only service-scoped EDGEZERO__SERVICES__<service_id>__* keys, and its own doc comment states "Legacy unscoped entries are not read." The manifests and one doc snippet were left on the unscoped form, so the logical-to-physical secret store mapping is inert and every Fastly request returns 500.

4 of the inline comments below carry a one-click GitHub suggestion — use Commit suggestion (or Add suggestion to batch) to apply them as commits on the PR branch. The remaining comments describe the fix in prose because the change spans multiple files or lines outside the diff.

How the blocking finding was confirmed

Built the debug wasm at this head and ran it under Viceroy against the harness's generated config:

Note the store name in the error is the logical trusted_server_secrets, not the mapped ts_secrets — the mapping never reached EnvConfig. No "edgezero_runtime_env not found" warning appeared, so the store opened fine; only the key form was wrong.

Effect on the CI gates: scripts/template-cache-local-test.sh esi fails 0 passed / 13 failed at this head (every request 500). With the one-line key change it is 21 passed / 0 failed, and inline mode is 8 passed / 0 failed.

Blocking

🔧 wrench

• edgezero_runtime_env mapping is inert under the v0.0.8 pin; every Fastly request 500s — see inline at fastly.toml:81
• Same inert key in the Fastly integration fixture — see inline at crates/trusted-server-integration-tests/fixtures/configs/viceroy-template.toml:89
• fastly.md documents the persisted mapping key in the form the runtime ignores — see inline at docs/guide/fastly.md:281

❓ question

• No migration path for configs that currently hold real secret values — see the cross-cutting section below

Non-blocking

♻️ refactor / 🤔 thinking / ⛏ nitpick

• Dead docs anchor #secret-store-migration — see inline at docs/guide/ec-setup-guide.md:41
• validate_config_for_deploy is identical to validate_config_for_startup; the resolved_secrets flag is dead — see inline at crates/trusted-server-core/src/integrations/datadome.rs:480
• First request's worker::Env is pinned for the isolate's lifetime — see inline at crates/trusted-server-adapter-cloudflare/src/app.rs:62
• access_token_secret is dropped silently while every sibling deprecated field warns — see inline at crates/trusted-server-core/src/settings.rs:1901
• Resolved DataDome server-side key is trimmed, but the resolved bypass credential deliberately is not — see inline at crates/trusted-server-core/src/integrations/datadome.rs:395
• Harness now appends secret entries the tip commit made redundant, with different values — see inline at scripts/template-cache-local-test.sh:226

👍 praise

• Disabled features never need their secret provisioned — see inline at crates/trusted-server-core/src/config_payload.rs:63

Cross-cutting / body-level findings

• ❓ No migration path for configs that currently hold real secret values. validate_settings_for_deploy no longer calls reject_placeholder_secrets, and secret-reference validation is only "non-empty after trim". An operator who pushes their existing trusted-server.toml — the one whose publisher.proxy_secret, ec.passphrase, and handlers[*].password hold real credential values — gets those values written into the config-store blob, which is exactly the exposure this change sets out to remove, and then a hard startup failure on the next deploy because no secret-store key by that name exists. The docs describe the greenfield ordering well but I could not find a "migrating an existing config" path. Is there an intended step this PR should carry: a ts config push warning when a referenced key is absent from the target secret store, a docs section, or a release note?

• 🤔 Per-request startup cost on Cloudflare and Spin. TrustedServerApp::routes() calls build_state() → load_startup_settings() on every request in both adapters, so each request now performs one secret-store read per declared secret field on top of the envelope parse and SHA-256 verify. Spin additionally opens and reads the KV config store per request, where it previously parsed a compile-time include_str! of trusted-server.example.toml. Fastly is inherently per-instance so it is unaffected. Worth confirming this is acceptable, or caching the built state per isolate.

• 📌 docs/guide/cli.md:69 still says "EdgeZero v0.0.4". docs/guide/configuration.md:191 was de-versioned in this PR ("The pinned EdgeZero loader…") for exactly this reason; cli.md keeps the literal version while Cargo.toml now pins v0.0.8. Not a changed file, so it cannot carry an inline comment.

• 📌 trusted-server.example.toml still ships deprecated selectors. Line 193 has # secret_store = "s3-auth" and line 511 has # credential_secret_store = "ts_secrets"; both fields are now deserialize-only, warn-and-ignore. The bypass block's comment also still says the credential "is loaded from the Secret Store at runtime (>= 32 bytes of high-entropy material)", which is now resolved at startup instead. Both lines fall outside this PR's diff hunks.

• 👍 The validation split is carefully built. Moving every secret check to a field-level #[validate(custom(...))] — including splitting validate_trusted_client_ip's struct-level schema validator so only the header checks remain there — is what lets validate_excluding_secrets strip exactly the secret leaves without dropping unrelated structural errors. The IntegrationSettings custom Debug that prints only integration ids closes a real leak, since resolved DataDome credentials now live in that raw JSON map. And the error paths never carry secret values, with tests asserting the negative.

CI Status

Only one check reported on head 4258a6bf:

• Analyze (javascript-typescript): PASS

Run Tests, Run Format, Integration Tests, and CodeQL's Rust analysis are not run on this head — their last runs were on the parent commit 71df1aa3, which is before the EdgeZero v0.0.8 bump that introduces the blocking finding. Branch protection reports no required checks on this branch. Results from running the full gate list locally against this head:

The two harness failures are the blocking finding, and both pass once it is fixed.